Explorations into Unsupervised Corpus Quality Assessment

Corpora, large bodies of text, are of great importance to the field of Natural Language Processing. They are for instance used to train systems on specific tasks through machine learning. When a system is trained on a corpus of low quality, it will not provide reliable results. We search for a metric of corpus quality by comparing the vocabulary growth data, the Zipf slopes and the Pareto exponents of corpora of different sizes, compositions and qualities in Dutch and English. Vocabulary growth curves are a great tool for directly spotting major deviations in a text. Our method applies a linear regression on these curves, which unfortunately does not capture the small errors that contaminate a corpus, like mistakes in spelling. The Zipf slope and Pareto exponent do show explicable deviations when the quality of a corpus changes. However, their standard values of 1 and 2 respectively, which are reported in much of the literature, also deviate for different corpus sizes. Consequently, we did not find a solid metric of corpus quality, but a crude measure can certainly be derived from our results.